A generative model for 2.5D vision: Estimating appearance, transformation, illumination, transparency and occlusion

Most vision research is based on some sort of a model of the world. The model is usually highly specific about one aspect of the world, for instance, the appearance of a human face, or the motion model for an object, but it does not model other parts of the scene. This makes models very brittle in any realistic application. We believe that in order to build more robust vision algorithms, models need to be capable of capturing different aspects of the data at the same time, be fairly simple, but adapt to the data. Flexible models, as defined in the machine learning community, are minimally structured probability models with a large number of parameters that can adapt so as to explain the input data. We describe a framework for designing and using flexible models for vision. The framework uses structured probability models to describe causes of variability in the data, variational methods for probabilistic inference and generalized expectation-maximization algorithm for parameter estimation. We show that within this framework, we can perform various vision tasks jointly, such as tracking, recognition, occlusion detection, object stabilization, object removal, filtering, illumination stabilization, panoramic image/video computation, artistic rendering, etc. In fact, we argue that dealing with these tasks jointly is easier than combining individually optimized modules in a typical engineering approach to signal processing.